## Motivation

The gVisor network stack is extensively employed in user-space tunneling software, often operating in promiscuous mode. In this configuration, the stack directly responds to all ICMPv4 Echo Request packets, irrespective of whether a [transport defaultHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L49) has already processed them. This behavior is often unintended in certain scenarios, as evidenced by issues such as:

- #8657
- containers/gvisor-tap-vsock#428
- xjasonlyu/tun2socks#361
- etc.

In these scenarios, users may prefer to utilize [SetTransportProtocolHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L517) to configure a custom `defaultHandler` for tailored processing of ICMPv4 Echo packets.

In issue #8657, @kevinGC proposed a potential solution:

```go
// If a customer ICMPv4 protocol handler has been set, use that in favor of default handling.
if p == header.ICMPv4ProtocolNumber {
    if _, ok := e.protocol.stack.transportProtocols[header.ICMPv4ProtocolNumber]; !ok {
        // handle the "normal" way
    }
}
```

However, this approach appears somewhat aggressive, as it could impair applications that depend on the existing gVisor stack behavior with ICMPv4 protocol handlers, such as [runsc itself](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/runsc/boot/loader.go#L1536). These programs would require code adjustments to accommodate this change, as shown below:

```go
// gvisor/runsc/boot/loader.go
func newEmptySandboxNetworkStack(clock tcpip.Clock, allowPacketEndpointWrite bool) (*netstack.Stack, error) {
    netProtos := []stack.NetworkProtocolFactory{ipv4.NewProtocol, ipv6.NewProtocol, arp.NewProtocol}
    transProtos := []stack.TransportProtocolFactory{
        tcp.NewProtocol,
        udp.NewProtocol,
        // icmp.NewProtocol4, runsc would need to remove this to allow stack ICMPv4Echo replies.
        icmp.NewProtocol6,
    }
    // ...
}
```

This patch introduces an alternative by enabling the stack to refrain from directly responding to ICMPv4 Echo packets delivered locally due to promiscuous mode, thereby allowing the defaultHandler to [handle](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/nic.go#L873) them independently.

**This proposal is presented as an initial step for discussion, and insights from experts on potential refinements or superior alternatives are warmly welcomed.**

**Testing and adjustments for ICMPv6 will be addressed once the approach is finalized.**

## Patch Details

In [ipv4.go:handleValidatedPacket](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/ipv4.go#L1108), the packet is evaluated based on `AcquireAssignedAddress` to determine local delivery or forwarding:

```go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

In promiscuous mode, packets destined for unknown addresses are assigned a temporary address and delivered locally:

```go
/*
handleValidatedPacket
 \----AcquireAssignedAddress
       \----AcquireAssignedAddressOrMatching
             \----addAndAcquireAddressLocked
*/
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB}, Temporary)
    // ...
}
```

By leveraging the `Temporary` field in [AddressProperties](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/registration.go#L492), we can identify packets delivered locally due to promiscuous mode. A new field, `LocalAddressTemporary`, is added to `NetworkPacketInfo` to record this status:

```go
// pkg/tcpip/stack/addressable_endpoint_state.go
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB, Temporary: true}, Temporary) // set AddressProperties.Temporary
    // ...
}

// pkg/tcpip/network/ipv4/ipv4.go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        pkt.NetworkPacketInfo.LocalAddressTemporary = addressEndpoint.Temporary() // packets delivered locally due to promiscuous mode
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

Finally, in [icmp.go:handleICMP](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/icmp.go#L282), direct replies to such packets are skipped:

```go
func (e *endpoint) handleICMP(pkt *stack.PacketBuffer) {
    // ...
    switch h.Type() {
    case header.ICMPv4Echo:
        // ...
        localAddressTemporary := pkt.NetworkPacketInfo.LocalAddressTemporary
        localAddressBroadcast := pkt.NetworkPacketInfo.LocalAddressBroadcast

        // It's possible that a raw socket or custom defaultHandler expects to
	// receive this packet.
        e.dispatcher.DeliverTransportPacket(header.ICMPv4ProtocolNumber, pkt)
        pkt = nil

        // Skip direct ICMP echo reply if the packet was received with a temporary
	// address, allowing custom handlers to take over.
        if localAddressTemporary {
            return
        }
        // ...
    }
}
```

## Quick Testing

To validate this patch, a simple test program and procedure are provided below:

```go
package main

import (
    "fmt"
    "net"
    "gvisor.dev/gvisor/pkg/tcpip"
    "gvisor.dev/gvisor/pkg/tcpip/header"
    "gvisor.dev/gvisor/pkg/tcpip/link/fdbased"
    "gvisor.dev/gvisor/pkg/tcpip/link/tun"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv4"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv6"
    "gvisor.dev/gvisor/pkg/tcpip/stack"
    "gvisor.dev/gvisor/pkg/tcpip/transport/icmp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/tcp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/udp"
)

func main() {
    fd, _ := tun.Open("tun0")
    ep, _ := fdbased.New(&fdbased.Options{
        FDs:                   []int{fd},
        MTU:                   1500,
        EthernetHeader:        false,
        PacketDispatchMode:    fdbased.Readv,
        MaxSyscallHeaderBytes: 0x00,
    })
    s := stack.New(stack.Options{
        NetworkProtocols: []stack.NetworkProtocolFactory{
            ipv4.NewProtocol,
            ipv6.NewProtocol,
        },
        TransportProtocols: []stack.TransportProtocolFactory{
            tcp.NewProtocol,
            udp.NewProtocol,
            icmp.NewProtocol4,
            icmp.NewProtocol6,
        },
    })

    nicID := s.NextNICID()
    s.CreateNICWithOptions(nicID, ep,
        stack.NICOptions{
            Disabled: false,
            QDisc:    nil,
        },
    )
    s.SetPromiscuousMode(nicID, true)
    s.SetSpoofing(nicID, true)
    s.SetRouteTable([]tcpip.Route{
        {Destination: header.IPv4EmptySubnet, NIC: nicID},
        {Destination: header.IPv6EmptySubnet, NIC: nicID},
    })
    s.SetTransportProtocolHandler(icmp.ProtocolNumber4, func(id stack.TransportEndpointID, pkt *stack.PacketBuffer) bool {
        h := header.ICMPv4(pkt.TransportHeader().Slice())
        fmt.Printf("icmpv4: %s->%s, hType: %v, ", id.RemoteAddress, id.LocalAddress, h.Type())
        if !pkt.NetworkPacketInfo.LocalAddressTemporary {
            fmt.Println("packet to permanent address, processed by stack")
            return true
        }
        fmt.Println("packet to temporary address, need to process by user")
        return true
    })
    protocolAddr := tcpip.ProtocolAddress{
        Protocol: ipv4.ProtocolNumber,
        AddressWithPrefix: tcpip.AddressWithPrefix{
            Address:   tcpip.AddrFromSlice(net.IPv4(11, 0, 0, 1).To4()),
            PrefixLen: 8,
        },
    }
    s.AddProtocolAddress(nicID, protocolAddr, stack.AddressProperties{PEB: stack.CanBePrimaryEndpoint, Temporary: false})

    fmt.Println("stack started ...")
    select {}
}
```

Test procedure:

```shell
$ ip tuntap add mode tun dev tun0
ip link set dev tun0 up

$ go run main.go
stack started ...

icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack

icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user

```

```shell
$ ping 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=64 time=0.441 ms
64 bytes from 11.0.0.1: icmp_seq=2 ttl=64 time=0.425 ms
64 bytes from 11.0.0.1: icmp_seq=3 ttl=64 time=0.437 ms
^C
--- 11.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 0.425/0.434/0.441/0.006 ms

$ ping 11.0.0.2
PING 11.0.0.2 (11.0.0.2) 56(84) bytes of data.
^C
--- 11.0.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2052ms
```

FUTURE_COPYBARA_INTEGRATE_REVIEW=#11609 from Amaindex:master 868dfbc
PiperOrigin-RevId: 753217427

## Motivation

The gVisor network stack is extensively employed in user-space tunneling software, often operating in promiscuous mode. In this configuration, the stack directly responds to all ICMPv4 Echo Request packets, irrespective of whether a [transport defaultHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L49) has already processed them. This behavior is often unintended in certain scenarios, as evidenced by issues such as:

- #8657
- containers/gvisor-tap-vsock#428
- xjasonlyu/tun2socks#361
- etc.

In these scenarios, users may prefer to utilize [SetTransportProtocolHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L517) to configure a custom `defaultHandler` for tailored processing of ICMPv4 Echo packets.

In issue #8657, @kevinGC proposed a potential solution:

```go
// If a customer ICMPv4 protocol handler has been set, use that in favor of default handling.
if p == header.ICMPv4ProtocolNumber {
    if _, ok := e.protocol.stack.transportProtocols[header.ICMPv4ProtocolNumber]; !ok {
        // handle the "normal" way
    }
}
```

However, this approach appears somewhat aggressive, as it could impair applications that depend on the existing gVisor stack behavior with ICMPv4 protocol handlers, such as [runsc itself](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/runsc/boot/loader.go#L1536). These programs would require code adjustments to accommodate this change, as shown below:

```go
// gvisor/runsc/boot/loader.go
func newEmptySandboxNetworkStack(clock tcpip.Clock, allowPacketEndpointWrite bool) (*netstack.Stack, error) {
    netProtos := []stack.NetworkProtocolFactory{ipv4.NewProtocol, ipv6.NewProtocol, arp.NewProtocol}
    transProtos := []stack.TransportProtocolFactory{
        tcp.NewProtocol,
        udp.NewProtocol,
        // icmp.NewProtocol4, runsc would need to remove this to allow stack ICMPv4Echo replies.
        icmp.NewProtocol6,
    }
    // ...
}
```

This patch introduces an alternative by enabling the stack to refrain from directly responding to ICMPv4 Echo packets delivered locally due to promiscuous mode, thereby allowing the defaultHandler to [handle](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/nic.go#L873) them independently.

**This proposal is presented as an initial step for discussion, and insights from experts on potential refinements or superior alternatives are warmly welcomed.**

**Testing and adjustments for ICMPv6 will be addressed once the approach is finalized.**

## Patch Details

In [ipv4.go:handleValidatedPacket](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/ipv4.go#L1108), the packet is evaluated based on `AcquireAssignedAddress` to determine local delivery or forwarding:

```go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

In promiscuous mode, packets destined for unknown addresses are assigned a temporary address and delivered locally:

```go
/*
handleValidatedPacket
 \----AcquireAssignedAddress
       \----AcquireAssignedAddressOrMatching
             \----addAndAcquireAddressLocked
*/
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB}, Temporary)
    // ...
}
```

By leveraging the `Temporary` field in [AddressProperties](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/registration.go#L492), we can identify packets delivered locally due to promiscuous mode. A new field, `LocalAddressTemporary`, is added to `NetworkPacketInfo` to record this status:

```go
// pkg/tcpip/stack/addressable_endpoint_state.go
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB, Temporary: true}, Temporary) // set AddressProperties.Temporary
    // ...
}

// pkg/tcpip/network/ipv4/ipv4.go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        pkt.NetworkPacketInfo.LocalAddressTemporary = addressEndpoint.Temporary() // packets delivered locally due to promiscuous mode
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

Finally, in [icmp.go:handleICMP](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/icmp.go#L282), direct replies to such packets are skipped:

```go
func (e *endpoint) handleICMP(pkt *stack.PacketBuffer) {
    // ...
    switch h.Type() {
    case header.ICMPv4Echo:
        // ...
        localAddressTemporary := pkt.NetworkPacketInfo.LocalAddressTemporary
        localAddressBroadcast := pkt.NetworkPacketInfo.LocalAddressBroadcast

        // It's possible that a raw socket or custom defaultHandler expects to
	// receive this packet.
        e.dispatcher.DeliverTransportPacket(header.ICMPv4ProtocolNumber, pkt)
        pkt = nil

        // Skip direct ICMP echo reply if the packet was received with a temporary
	// address, allowing custom handlers to take over.
        if localAddressTemporary {
            return
        }
        // ...
    }
}
```

## Quick Testing

To validate this patch, a simple test program and procedure are provided below:

```go
package main

import (
    "fmt"
    "net"
    "gvisor.dev/gvisor/pkg/tcpip"
    "gvisor.dev/gvisor/pkg/tcpip/header"
    "gvisor.dev/gvisor/pkg/tcpip/link/fdbased"
    "gvisor.dev/gvisor/pkg/tcpip/link/tun"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv4"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv6"
    "gvisor.dev/gvisor/pkg/tcpip/stack"
    "gvisor.dev/gvisor/pkg/tcpip/transport/icmp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/tcp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/udp"
)

func main() {
    fd, _ := tun.Open("tun0")
    ep, _ := fdbased.New(&fdbased.Options{
        FDs:                   []int{fd},
        MTU:                   1500,
        EthernetHeader:        false,
        PacketDispatchMode:    fdbased.Readv,
        MaxSyscallHeaderBytes: 0x00,
    })
    s := stack.New(stack.Options{
        NetworkProtocols: []stack.NetworkProtocolFactory{
            ipv4.NewProtocol,
            ipv6.NewProtocol,
        },
        TransportProtocols: []stack.TransportProtocolFactory{
            tcp.NewProtocol,
            udp.NewProtocol,
            icmp.NewProtocol4,
            icmp.NewProtocol6,
        },
    })

    nicID := s.NextNICID()
    s.CreateNICWithOptions(nicID, ep,
        stack.NICOptions{
            Disabled: false,
            QDisc:    nil,
        },
    )
    s.SetPromiscuousMode(nicID, true)
    s.SetSpoofing(nicID, true)
    s.SetRouteTable([]tcpip.Route{
        {Destination: header.IPv4EmptySubnet, NIC: nicID},
        {Destination: header.IPv6EmptySubnet, NIC: nicID},
    })
    s.SetTransportProtocolHandler(icmp.ProtocolNumber4, func(id stack.TransportEndpointID, pkt *stack.PacketBuffer) bool {
        h := header.ICMPv4(pkt.TransportHeader().Slice())
        fmt.Printf("icmpv4: %s->%s, hType: %v, ", id.RemoteAddress, id.LocalAddress, h.Type())
        if !pkt.NetworkPacketInfo.LocalAddressTemporary {
            fmt.Println("packet to permanent address, processed by stack")
            return true
        }
        fmt.Println("packet to temporary address, need to process by user")
        return true
    })
    protocolAddr := tcpip.ProtocolAddress{
        Protocol: ipv4.ProtocolNumber,
        AddressWithPrefix: tcpip.AddressWithPrefix{
            Address:   tcpip.AddrFromSlice(net.IPv4(11, 0, 0, 1).To4()),
            PrefixLen: 8,
        },
    }
    s.AddProtocolAddress(nicID, protocolAddr, stack.AddressProperties{PEB: stack.CanBePrimaryEndpoint, Temporary: false})

    fmt.Println("stack started ...")
    select {}
}
```

Test procedure:

```shell
$ ip tuntap add mode tun dev tun0
ip link set dev tun0 up

$ go run main.go
stack started ...

icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack

icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user

```

```shell
$ ping 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=64 time=0.441 ms
64 bytes from 11.0.0.1: icmp_seq=2 ttl=64 time=0.425 ms
64 bytes from 11.0.0.1: icmp_seq=3 ttl=64 time=0.437 ms
^C
--- 11.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 0.425/0.434/0.441/0.006 ms

$ ping 11.0.0.2
PING 11.0.0.2 (11.0.0.2) 56(84) bytes of data.
^C
--- 11.0.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2052ms
```

FUTURE_COPYBARA_INTEGRATE_REVIEW=#11609 from Amaindex:master 868dfbc
PiperOrigin-RevId: 753217427

## Motivation

The gVisor network stack is extensively employed in user-space tunneling software, often operating in promiscuous mode. In this configuration, the stack directly responds to all ICMPv4 Echo Request packets, irrespective of whether a [transport defaultHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L49) has already processed them. This behavior is often unintended in certain scenarios, as evidenced by issues such as:

- #8657
- containers/gvisor-tap-vsock#428
- xjasonlyu/tun2socks#361
- etc.

In these scenarios, users may prefer to utilize [SetTransportProtocolHandler](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/stack.go#L517) to configure a custom `defaultHandler` for tailored processing of ICMPv4 Echo packets.

In issue #8657, @kevinGC proposed a potential solution:

```go
// If a customer ICMPv4 protocol handler has been set, use that in favor of default handling.
if p == header.ICMPv4ProtocolNumber {
    if _, ok := e.protocol.stack.transportProtocols[header.ICMPv4ProtocolNumber]; !ok {
        // handle the "normal" way
    }
}
```

However, this approach appears somewhat aggressive, as it could impair applications that depend on the existing gVisor stack behavior with ICMPv4 protocol handlers, such as [runsc itself](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/runsc/boot/loader.go#L1536). These programs would require code adjustments to accommodate this change, as shown below:

```go
// gvisor/runsc/boot/loader.go
func newEmptySandboxNetworkStack(clock tcpip.Clock, allowPacketEndpointWrite bool) (*netstack.Stack, error) {
    netProtos := []stack.NetworkProtocolFactory{ipv4.NewProtocol, ipv6.NewProtocol, arp.NewProtocol}
    transProtos := []stack.TransportProtocolFactory{
        tcp.NewProtocol,
        udp.NewProtocol,
        // icmp.NewProtocol4, runsc would need to remove this to allow stack ICMPv4Echo replies.
        icmp.NewProtocol6,
    }
    // ...
}
```

This patch introduces an alternative by enabling the stack to refrain from directly responding to ICMPv4 Echo packets delivered locally due to promiscuous mode, thereby allowing the defaultHandler to [handle](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/nic.go#L873) them independently.

**This proposal is presented as an initial step for discussion, and insights from experts on potential refinements or superior alternatives are warmly welcomed.**

**Testing and adjustments for ICMPv6 will be addressed once the approach is finalized.**

## Patch Details

In [ipv4.go:handleValidatedPacket](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/ipv4.go#L1108), the packet is evaluated based on `AcquireAssignedAddress` to determine local delivery or forwarding:

```go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

In promiscuous mode, packets destined for unknown addresses are assigned a temporary address and delivered locally:

```go
/*
handleValidatedPacket
 \----AcquireAssignedAddress
       \----AcquireAssignedAddressOrMatching
             \----addAndAcquireAddressLocked
*/
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB}, Temporary)
    // ...
}
```

By leveraging the `Temporary` field in [AddressProperties](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/stack/registration.go#L492), we can identify packets delivered locally due to promiscuous mode. A new field, `LocalAddressTemporary`, is added to `NetworkPacketInfo` to record this status:

```go
// pkg/tcpip/stack/addressable_endpoint_state.go
func (a *AddressableEndpointState) AcquireAssignedAddressOrMatching(localAddr tcpip.Address, f func(AddressEndpoint) bool, allowTemp bool, tempPEB PrimaryEndpointBehavior, readOnly bool) AddressEndpoint {
    // ...
    if !allowTemp { // e.nic.Promiscuous()
        return nil
    }
    // ...
    ep, err := a.addAndAcquireAddressLocked(addr, AddressProperties{PEB: tempPEB, Temporary: true}, Temporary) // set AddressProperties.Temporary
    // ...
}

// pkg/tcpip/network/ipv4/ipv4.go
func (e *endpoint) handleValidatedPacket(h header.IPv4, pkt *stack.PacketBuffer, inNICName string) {
    // ...
    if addressEndpoint := e.AcquireAssignedAddress(dstAddr, e.nic.Promiscuous(), stack.CanBePrimaryEndpoint, true /* readOnly */); addressEndpoint != nil {
        pkt.NetworkPacketInfo.LocalAddressTemporary = addressEndpoint.Temporary() // packets delivered locally due to promiscuous mode
        subnet := addressEndpoint.AddressWithPrefix().Subnet()
        pkt.NetworkPacketInfo.LocalAddressBroadcast = subnet.IsBroadcast(dstAddr) || dstAddr == header.IPv4Broadcast
        e.deliverPacketLocally(h, pkt, inNICName)
    } else if e.Forwarding() {
        e.handleForwardingError(e.forwardUnicastPacket(pkt))
    } else {
        stats.ip.InvalidDestinationAddressesReceived.Increment()
    }
}
```

Finally, in [icmp.go:handleICMP](https://github.com/google/gvisor/blob/6b2bcc44e061f48c6dd9cc6048ae17d389a2f22d/pkg/tcpip/network/ipv4/icmp.go#L282), direct replies to such packets are skipped:

```go
func (e *endpoint) handleICMP(pkt *stack.PacketBuffer) {
    // ...
    switch h.Type() {
    case header.ICMPv4Echo:
        // ...
        localAddressTemporary := pkt.NetworkPacketInfo.LocalAddressTemporary
        localAddressBroadcast := pkt.NetworkPacketInfo.LocalAddressBroadcast

        // It's possible that a raw socket or custom defaultHandler expects to
	// receive this packet.
        e.dispatcher.DeliverTransportPacket(header.ICMPv4ProtocolNumber, pkt)
        pkt = nil

        // Skip direct ICMP echo reply if the packet was received with a temporary
	// address, allowing custom handlers to take over.
        if localAddressTemporary {
            return
        }
        // ...
    }
}
```

## Quick Testing

To validate this patch, a simple test program and procedure are provided below:

```go
package main

import (
    "fmt"
    "net"
    "gvisor.dev/gvisor/pkg/tcpip"
    "gvisor.dev/gvisor/pkg/tcpip/header"
    "gvisor.dev/gvisor/pkg/tcpip/link/fdbased"
    "gvisor.dev/gvisor/pkg/tcpip/link/tun"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv4"
    "gvisor.dev/gvisor/pkg/tcpip/network/ipv6"
    "gvisor.dev/gvisor/pkg/tcpip/stack"
    "gvisor.dev/gvisor/pkg/tcpip/transport/icmp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/tcp"
    "gvisor.dev/gvisor/pkg/tcpip/transport/udp"
)

func main() {
    fd, _ := tun.Open("tun0")
    ep, _ := fdbased.New(&fdbased.Options{
        FDs:                   []int{fd},
        MTU:                   1500,
        EthernetHeader:        false,
        PacketDispatchMode:    fdbased.Readv,
        MaxSyscallHeaderBytes: 0x00,
    })
    s := stack.New(stack.Options{
        NetworkProtocols: []stack.NetworkProtocolFactory{
            ipv4.NewProtocol,
            ipv6.NewProtocol,
        },
        TransportProtocols: []stack.TransportProtocolFactory{
            tcp.NewProtocol,
            udp.NewProtocol,
            icmp.NewProtocol4,
            icmp.NewProtocol6,
        },
    })

    nicID := s.NextNICID()
    s.CreateNICWithOptions(nicID, ep,
        stack.NICOptions{
            Disabled: false,
            QDisc:    nil,
        },
    )
    s.SetPromiscuousMode(nicID, true)
    s.SetSpoofing(nicID, true)
    s.SetRouteTable([]tcpip.Route{
        {Destination: header.IPv4EmptySubnet, NIC: nicID},
        {Destination: header.IPv6EmptySubnet, NIC: nicID},
    })
    s.SetTransportProtocolHandler(icmp.ProtocolNumber4, func(id stack.TransportEndpointID, pkt *stack.PacketBuffer) bool {
        h := header.ICMPv4(pkt.TransportHeader().Slice())
        fmt.Printf("icmpv4: %s->%s, hType: %v, ", id.RemoteAddress, id.LocalAddress, h.Type())
        if !pkt.NetworkPacketInfo.LocalAddressTemporary {
            fmt.Println("packet to permanent address, processed by stack")
            return true
        }
        fmt.Println("packet to temporary address, need to process by user")
        return true
    })
    protocolAddr := tcpip.ProtocolAddress{
        Protocol: ipv4.ProtocolNumber,
        AddressWithPrefix: tcpip.AddressWithPrefix{
            Address:   tcpip.AddrFromSlice(net.IPv4(11, 0, 0, 1).To4()),
            PrefixLen: 8,
        },
    }
    s.AddProtocolAddress(nicID, protocolAddr, stack.AddressProperties{PEB: stack.CanBePrimaryEndpoint, Temporary: false})

    fmt.Println("stack started ...")
    select {}
}
```

Test procedure:

```shell
$ ip tuntap add mode tun dev tun0
ip link set dev tun0 up

$ go run main.go
stack started ...

icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack
icmpv4: 10.161.22.19->11.0.0.1, hType: 8, packet to permanent address, processed by stack

icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user
icmpv4: 10.161.22.19->11.0.0.2, hType: 8, packet to temporary address, need to process by user

```

```shell
$ ping 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=64 time=0.441 ms
64 bytes from 11.0.0.1: icmp_seq=2 ttl=64 time=0.425 ms
64 bytes from 11.0.0.1: icmp_seq=3 ttl=64 time=0.437 ms
^C
--- 11.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2012ms
rtt min/avg/max/mdev = 0.425/0.434/0.441/0.006 ms

$ ping 11.0.0.2
PING 11.0.0.2 (11.0.0.2) 56(84) bytes of data.
^C
--- 11.0.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2052ms
```

FUTURE_COPYBARA_INTEGRATE_REVIEW=#11609 from Amaindex:master 868dfbc
PiperOrigin-RevId: 753217427